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ABSTRACT 



An evaluation model was developed to aid the 
classroom instructor in more effectively teaching his course by 
providing him with the basic data upon which to make judgements about 
student abilities, course attitudes, grade distributions, test 
appropriateness, etc. The MERMAC system was designed so that each 
instructor could obtain the data in a usable format and also be able 
to provide relatively immediate feedback to the students taking an 
examination or filling out a course evaluation questionnaire. A 
description of the software system and its interface to the 
evaluation model is presented. (Author) 
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Vos tract 



An evaluation model was developed to aid the classroom instructor 
in more effectively teaching bis course by providing Jiitn with the basic 
data upon which to make judgements about student abilities, course 
attitudes, grade distributions, test appropriateness, etc. The MESMAC 
system was designed so that each instructor could obtain the data in a 
usable format and also be able to provide relatively immediate feedback to 
the studeuts taking an examination or filling out a course evaluation 
questionnaire. A description of the software system and its interface 
to the evaluation model is presented. 



MhRMAC: A Model and System for Test 

and Questionnaire Analysis*" 

Lawrence M. Aleamoni 

We know that what is taught in the classroom may vary from excellent 
to poor, applicable to useless, and underst 3 to misunderstood, and 
that instruction exists if you have an instructor, a textbook, a TV pre- 
sentation, or a film presentation. What is LEARNED, however, is not 
necessarily that which is taught. Therefore, in order to determine the 
extent, type, and degree of learning in a classroom, some method of 
evaluation is necessary. 

Let us begin with the premise that if instruction is to be effective, 
then there must be some interaction between what the instructor presents 
and what the students comprehend. Therefore, in order to aid the in- 
structor in improving his instruction, e course evaluator must concentrate 
on one major aspect of the instructor-student interaction, which is the 
knowledge of student learning. This knowledge should be supplied to 
(a) the student so that he may be constantly aware of his progress in 
comparison to others in his class and in comparison to course standards 
and (b) the instructor so that he may alter his instructional rate, 
content, or method in order to suit the observed learning. 

What the student generally studies and learns tends to be the material 
that is tested, not necessarily the material that is presented. Tfests 
used in a course represent, in a very practical manner, the direction an 
instructor thinks the students should go which, in turn, determines the 
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emphasis that the students will place on the material treated by the 
test. Good tests can, therefore, orient student learning., while the develop- 
ment of the tests and the knowledge of student responses... can help to 
improve instruction. 



Description 

The bases, therefore, for developing a model and system for test- 
analysis should be to: 

1, assist the instructor in developing valid and reliable tests; and 

2. provide rapid and meaningful feedback to the instructor and stu- 
dents . 

MERMAC was developed with the above bases in mind. It is made up of 
two sets of programs (see Figure 1) : (a.) utility (data manipulation) 

programs, and (b) test and q ?, est? r, ”‘ lysis programs The oeveu 

utility programs allow the user to copy, edit, match, merge, sequence, 
sortt, and recode the input data. Generally, the purpose of these programs 
is to prepare the dat^a for input to the test and questionnaire analysis 
programs. The six te&jt and questionnaire programs allow the user to: 

1, Score item data and produce up to forty subscores for each 

individual . Each item amd response may be weighted to arrive at 
the scores. Any item may be included in more than one subscore 
and be weighted differently in each. An example of tl ^ SCfiiRE 
program output is presented in Figure 2. 
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MERf-IAC System 



Data Input 



Data Manipulation 




Test and Questionnaire Analysis 




SC0RE 


T0TAL 


ERR0R 


ITEM 


SELECT 


QUEST 




Fig. 1. The MEKKAC system. 
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1C 


8 


37 


330323181norregaard 






6 


e 


28 


7778 88888 owen 








12 


46 


357U17UU5owens 






i 


9 


33 


331 448266 pearson 






11 


J 


322U07590pl sko 






5 




32 


1111 IOOCOplutchak 






6 


5 


28 


343441821rodell 






1C 


1U 


50 


340448994rothbaum 
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6 


9 
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Sample O oSr TJ 5v0R£ 







2. Take scores for a group of individuals at*' 1 produce - frequency 
distribution and histogram, mean, median, standard deviation, 
Kuder-RichardBon reliability, /standard error of measurement, 
and Spearman-Brown prophesy for a reliability of .90. In addition, 
individual raw scores, standard scores, and percentiles may be 
listed. Individual raw scores and standard scores can be weighted, 
summed, and the sum assigned a letter grade. All this data can 
be easily provided to the student. An example of the T0TAL 
^program -is preaented in Figure 3. 
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SUMMARY OF TEST STAT 1 


ST ICS 








TEST FREQUENCY DISTRIBl 


Number of Items 


15 


Raw 


Standard 


Per 






Cum 






Score 


Score 


Centile Percent 


Freq 


freq 


Mean Score 


3.65 


14 


931 


99 


0.6# 


1 


173 


Median Score 


3.23 


13 


889 


99 


1 .2# 


2 


172 






12 


848 


98 


0.0# 


0 


170 


Standard Deviation 


2.40 


11 


806 


98 


1 .7# 


3 


170 






10 


754 


97 


0.6# 


1 


167 


Reliability (KR-21) 


0.558 


9 


723 


96 


0,6# 


1 


166 






8 


681 


95 


0.6# 


1 


166 


S.E* of Measurement 


1 <60 


7 


639 


95 


4 .Q# 


7 


164 






6 


593 


91 


5.2# 


9 


157 


Possible Low Score 


0 


5 


556 


86 


10,4# 


18 


148 


Possible High Score 


15 


4 


514 


75 


19.1# 


33 


13C 






3 


473 


56 


22.5# 


39 


97 


Obtained Low Score 


0 


2 


431 


34 


20 .S# 


36 


58 


Obtained High Score 


14 


1 


389 


13 


10.4# 


18 


22 






0 


348 


2 


2.3# 


4 


4 


Number of Scores 


173 














Blank Scores 


0 














Invali d Scores 


0 














Valid Scores 


173 















Each f Re presents ^ Person.. 



# 




M Spearman-Brown Prophesy For- 
mula: In order for this test 

TO OBTAIN A RELIABILITY OF *90 
IT MOST BE 6.14 LONGER. {92 
ADDITIONAL I TEMS ) .*' 



TOTAL TEST PROGRAM USING SUNWING AND GRADING 

Section 063 



Part-One 
Raw Stand Per 
361 443742 anderberg 3 371 23 

Grade is E 



Part-Two Part* Three Part-Four Total Summed Std Score 

Raw Stand Pct Raw Stand Per Raw Stand Per Raw Stand Pct Raw Stand Per 
4 475 SI 2 365 18 3 405 39 12 322 7 39 328 7 



Figu££ 3. Sample output from T$TAL 
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3. Return to each student a page containing his taa^-score and a 
list of the items he missed with his responses and the correct 
responses. An example of the ERR0R program output is presented 
in Figure 4* 

STUDENT ERROP REPORT 









Name 






Test Score 










Anderson 






11 










*** Here 


. S A SUMKARY 


or 


THE U !TEM(s) YOU MISSED*** 




ITEM 


Your 


Correct 


Item Your 


Correct 


Item 


Your Correct 1 .'em Your 


Correct 


No, 


Response 


Response 


No. Response 


Response 


NO. 


Response Response No. Response 


Response 


6 . 


C 


D 


7. D 


C 


12. 


A C 15. D 


C 



Figure 4. Sample output. jfxom.-ERR^R 



4. Analyze his item data by providing a plot of the.. percentage of 

individuals responding to the keyed response by fifths of the total 
score distribution. For each item alternative the proportion 
of individuals responding, a point biserial correlation, and 
the number responding to each alternative by fifths is provided. 

An example of the ITEM program output is presented in Figure 5. 



ITEM ANALYSIS OUTPUT 



ITEM 4 


PERCENT OF 


CORRECT 


RESPONSE BY FIFTHS 


MATRIX OF 


RESPONSES BY FIFTHS 


D IS 


C0R^:CT RESPONSE 












A 


B 


C D 




OMlT 


1st + 






• 


1st 


1 


2 


0 39 


1 


6 


2nd + 






« 


2nd 


3 


i 


2 22 


5 


0 


3rd + 






* 


3rd 


6 


1 


1 26 


6 


J 


liTH + 




* 




Vth 


11 


0 


2 17 


6 


0 


5th + 


* 

-r-h -f- 






5 th 


u 


1 


5 6 


6 


0 


0 10 


20 30 


■ ** T " * * • p i > * 

40 50 


60 70 80 90 <1 00 


PROP 


0.14 


0.03 


0.06 0.6U 


O.lll 


0.00 










RPBI 


-0.21 


0.01 


-0,13. -0.37 -0*19 


0.00 
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Figure 5. Sample output from ITEM" 
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5. Analyse his item data by using some external criterion rather 
than the keyed test score. TV.e output format of SELECT is 
identical to that of ITEM. 

6. Summarise item data from questionnaires or tests with no known 
correct answers by providing a frequency distribution of responses, 
a weighted mean, and a standard deviation for each item. In 
addition, sufcscores may be generated with means, standard 
deviation, split-half reliabilities, and percentage of individuals 
responding to the contributing items. It if also possible to 

s 

assign d( 'lies to the item and sr.bscore means based on a table 
look-up. An example of the QlJEST program output is presented 
ir "Hgure 6 . 



QUEST ANALYSIS OUTPUT 
Sex Distribution 

FEMALE MALE OMIT 

\ 0.13 0.63 0.24 



EXPECTCO GRADE 

A 8 C 0 t OMIT 
0.19 0.25 0.25 0.06 0.00 0.25 



ITEM 


SA 


A 


D 


so 


OMIT 


BEST MEAN 


3.0. DECL 


0123456789 


1. 


0.25 


o.44 


0.06 


0.13 


0.13 


3 A 


2.93 


0.95 


4 


© 


2. 


0.13 


0.00 


0.38 


0.35 


0.13 


30 


3.14 


0.68 


7 




3 . 


0.06 


0.50 


0,19 


0.13 


0.13 


3A 


2.57 


1 .01 


5 


n 


V . 


0.06 


a.44 


0.25 


0*13 


0.13 


SA 


2.50 


0.80 


3 


» - 


5 . 


0.56 


0.25 


0.06 


6t00 


0.13 


SA 


3.57 


0.70 


8 


* 


6. 


0.38 


0.31 


0.13 


0.06 


0.13 


SA 


3.14 


0.85 


6 


* 


7 . 


0.44 


0.25 


0.06 


0.00 


0.25 


SA 


3.50 


0.81 


9 


• 


8. 


0.06 


0 Jill 


0.25 


0.06 


0,19 


SA 


2.38 


0.80 


6 


* 


9 . 


0.19 


0.44 


0.13 


0.06 


0.19 


SA 


2.92 


0.70 


6 


• 


10 . 


0.00 


0.19 


0.63 


0.06 


0,13 


30 


2.86 


0.89 


6 


« 






•'-SUBSCORE-'* 


ITE??S RESP 


MEAN 3 


.O. RCL 


N-01 




Total 




10 


0 


.85 


2.95 0 


.68 0.92 


6 






PART- ONE 


5 


0 


.88 


2.94 0 


.91 0.98 


5 






PART- TWO 


5 


0 


.82 


2.95 0 


.85 0.90 


6 





SAMPLE SIZE = 16 
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Figure 6 



Sample output fro" 1 QUEST 
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The MERMAC system is written in Basic Assembly Language (BAL) for 
XSM System/ 360 models 40 and above which have Operating System ,{0S) with 
Queued Sequential Access Method (QSAM) support. 



Application 
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Let us assume that the first basis of assisting the instructor in 
developing valid and reliable tests is also the basis for developing an 
evaluation model* The first step in the model, therefore, must be to 
provide assistance to the instructor in the determination of instructional: 
objectives and the appropriate methods of measuring those objectives. 

This should also involve helping the instructor to write and select his 
test items. Figure 7 presents the interface of the model and the system. 

The first step in the process which involves MERMAC would be to take 
the newly constructed or previously used items and administer them to 
the class and then subject them to an item analysis using either the keyed! 
score or an external criterion score. Through the use of the item 
analysis statistics and the content knowledge of the instructor, a reliable 
and valid set of question s can be generated. 

Once the final set of questions has been obtained, then the in- 
structor is ready to have them administered, scored* and analyzed to 
determine what number of subscores exist or should be constructed* The 
instructor could now expect to receive score distributions on the total 
test or subscores for students in bis course. These score distributions 
would allow him to determine how well the students were accomplishing his 
course objectives and what grades should be assigned. At the same time, 
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EVALUATION MODEL 



MERMAC APPLICATION 



Edification of 
Course and Objec- 

TIVES 



A 



7pr 



Determina- 
tion of 
Course Ob- 
jective 
Accomplish- 
ment 



Developing and Deter- 
mining Instructional 
Objectives 



Developing Instru- 

% MENTS TO MEASURE 

Objectives 



> 

Employ the 
ments to Me 
Student Ace 


^ 

Instru- 

iASURE 

IIEVEMENT 


V 


l 


Provide Feedback to 
. the Instructor and 
Students 


V 




Summarize Data During 
the Term for Final 
^rade Determination 



Student Course Evaluation 
Data Obtained and Fed 
Back to the Instructor 






ITEM 




. SELECT 













SCORE 



TOTAL 



ITEM 



TOTAL 



OUEST 
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Fig. 7. Interface of the evaluation model and HERJIAC. 
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the i tudents can be made aware of their performance on specific questions 
and in comparison to their classmates through the Student Error Reports. 

The instructor should discuss with the students those items most 
frequently missed and then provide them with a mimeographed sheet out- 
lining the topical areas covered by the test questions showing how they 
fit into his scheme or course objectives . By allowing the students only to 
take the mimeographed sheet ale np wi m their Student Error Reports, the 
instructor is abLe to help the srud^-ts identify their weakest areas and 
concentrate on improving them rather than memorizing the correct answer to 
each specific question posed. Xn edition, this allows the instructor to 
begin building a secure, reliable, and valid item bank for use later in 
that course or in other courses. 

Once an instructor has developed the objective tests, term papers, 
laboratory exercises, etc., to be used in his course, he can now use the 
summing and grading option of the T0TAL program to accumulate and weight 
the scores and grades he assigns. This not only relieves the instructor 
of the time-consuming clerical operations involved but also gives him a 
more objective method of combining the scores and grades assigned. In addi» 
tion, each student can be provided with his complete record of performance 
during the course along with his final grade. 

In order to satisfy the second basis of returning results to the instruc- 
tor and students in the fastest possible time, particular types of hardware 
and support staff are needed. The hardware should consist of a keypunch, an 
optical scanner, and either an IBM 360 system model 40 or above or IBM 370 
system model 135 or above. The support staff could consist of only in- 
dividuals capable of operating the keypunch and optical scanner, assuming 
that each institution has a staff available to maintain the computer operation. 
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The only part of the proposed model that has not been dealt with is 
the Questionnaire Analysis. The QUEST program provides a means of col- 
lecting student attitudes and opinions toward a course and instructor , 
analyzing them, comparing the results to normari\e and then returning 

the results to the instructor. An example of this wrcald b-= the Illinois 
Course Evaluation Questionnaire (CSQ) (Spencer 6c Al^omoni L970) which 
yields ’responses to 50 items and six subscores (gzans-ral course attitude, 
method of instruction, course content, interest-arte^rion. instructor, and 
specific items) . The subscore can then be compared nor?- developed for 
different courses, course levels, departments, colleges, rank of instructor, 
etc. The instructor can also obtain results on any items that he may wish 
to generate. 

Questionnaire data, in particular course evaluation data, can also 
help the instructor in diagnosing his course for possible changes in the 
(a) method of instruction, (b) course content, (c) personal delivery, 

(d) types of tests, (e) instructional materials, etc. 



Summary 



The MERMAC system, therefore, can provide a course evaluator and 
the instructor with various types of data that would be helpful in im- 
proving his course. The other essential elements needed to ensure that the 
data is effectively utilized in improving any course is a qualified, moti- 
vated course evaluator and an interested, willing instructor. 



O 

ERIC 



13 



- 12 - 



Reference 



Spencer* R« E, & Aleamoni, L* M. A student course evaluation c 
Journal of Educational Measurement , 1970, 2(3), 209-210. 




astionnaire . 



14 



